Rating Whole-Body Suspiciousness Factors in Automated Surveillance of a Public Area

Abstract We tested clues to automated detection of suspicious behavior of pedestrians in image sequences of 71,236 images in 106 sequences, developed at the U.S. Army Research Laboratory for testing surveillance systems. We achieved a 66% success rate at identifying suspicious behavior on infrared video and 64% on color video using just clues obtained from the overall body motion without attempting to classify further what people were doing. We used seven clues including the norm of the acceleration vector as measured at different granularities, atypicality of location, speed, atypicality of the velocity vector with historical data, relative size of the object, length of the track, and "contagion" from nearby suspicious tracks. The acceleration norm averaged over different time scales significantly outperformed the other clues on the dataset, and our conclusion is that it should be the primary focus of systems assessing suspicious behavior in video.